In order to provide a better experience to software application users, developers might update or make changes to software applications by adding/changing features or modifying designs. When deployed to users, however, those updates or changes might not achieve the initially intended goal. For example, a change in the difficulty level of a game application designed to engage a user more in the game might lead to a reduction in the amount of time a user spends playing the game, which may be because the user finds the updated game too hard or too simple to play. To reduce the uncertainty of the impact that updates to a software application may have on users, experiments on the software application, such as “A/B” testing, may be employed to test the effects of changes to a software application on a group of users before applying the changes to the software application to all users.
Most of the existing tools for software application experiments support comparisons between two versions of a software application feature: a control version/variation, referring to the original version of the feature, and a test version/variation, referring to the new version of the feature. While these tools might be able to satisfy the needs for software application experiments where only two versions/variations of a feature are involved, they become less efficient when the number of variations of software application features gets larger. For example, to test more than two variations of a software application feature, the developer might have to run a software application experiment tool multiple times and manually select the variations to be tested each time. When the number of variations gets larger, the process might become even more cumbersome and time-consuming for the software application developer.
The disclosure made herein is presented with respect to these and other considerations.